Skip to content

Conversation

@kmruiz
Copy link
Collaborator

@kmruiz kmruiz commented Oct 16, 2025

Proposed changes

Checklist

Also remove the embedding provider interface as it will be covered
in another PR
@kmruiz kmruiz changed the title chore: integration with embeddings API MCP-245 chore: upgrade vercel sdk to latest version API MCP-245 Oct 16, 2025
@kmruiz kmruiz marked this pull request as ready for review October 16, 2025 14:40
@kmruiz kmruiz requested a review from a team as a code owner October 16, 2025 14:40
Copilot AI review requested due to automatic review settings October 16, 2025 14:40
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR upgrades the Vercel AI SDK to version 5.x and updates related Azure SDK packages to version 2.x. The changes maintain compatibility with the new SDK API while removing the Ollama provider dependency.

  • Updated AI SDK types from LanguageModelV1 to LanguageModel
  • Migrated to new Vercel AI SDK API patterns (stepCountIs, token usage property changes)
  • Removed Ollama provider and associated code

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/accuracy/sdk/models.ts Updates type imports and removes Ollama model implementation
tests/accuracy/sdk/agent.ts Migrates to new AI SDK API with updated token usage properties
tests/accuracy/sdk/accuracyTestingClient.ts Adds type imports and fixes function signature compatibility
package.json Upgrades AI SDK packages to v5.x and Azure SDK packages to v2.x
.github/workflows/accuracy-tests.yml Updates Azure OpenAI API URL environment variable

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

📊 Accuracy Test Results

📈 Summary

Metric Value
Commit SHA b0c6c6261347048eb0425f93863eb081345153e7
Run ID 5032354b-6122-4d20-833d-cb973738f554
Status done
Total Prompts Evaluated 75
Models Tested 1
Average Accuracy 90.3%
Responses with 0% Accuracy 7
Responses with 75% Accuracy 1
Responses with 100% Accuracy 67

📊 Baseline Comparison

Metric Value
Baseline Commit dce19d8251d70eb14db981c38a357d24ac5c9d83
Baseline Run ID a7282221-3260-45c5-a96d-67dfd3519e49
Baseline Run Status done
Responses Improved 1
Responses Regressed 5

📎 Download Full HTML Report - Look for the accuracy-test-summary artifact for detailed results.

Report generated on: 10/16/2025, 3:47:38 PM

@kmruiz kmruiz merged commit 3b000f5 into main Oct 16, 2025
28 of 33 checks passed
@kmruiz kmruiz deleted the chore/mcp-245 branch October 16, 2025 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants